E Study of Statistical Intwitions*

Stan ford University The study of intuitions and errors in judgment un~‘er umwtainty is complicated by several factors: discrepancies between acceptance and application of normative rules: effects of content on me application of rules; Sucratic hints that create intuitions while testing them; demand characteristics of within-subject experiments; subjects’ interpretations of experimental messages according to standard conversational rules. The positive analysis of a iudgmental error in terms of heuri.stics may be supplemented by a negative analysis, which seeks to explain why the correct rule is not intuitively compelling. A negative analysis of non-regressive prediction is outlined. Much of the recent literature on judgment and, inductive reasoning has been concerned with errors, biases and fallacies in a variety of mental tasks (see, e.g., Einhorn and Hogarth, 1981; Hammond, McClelland and Mumpower, I980; Kahneman, Slavic, and Tversky, in press; Nisbett and Ross, 1980; Shweder, 1980; Slavic, Fishhoff and Lichtenstein, 1977; Tversky and Kahneman, 1974). The bmphasis on the study of errors is characteristic of research in human judgment, but is not unique to this domain: we use illusions to understand the principles of normal perception and we learn about memory by studying forgetting. Errors of reasoning, however, are unique among cognitive failures in two significant respects: they are somewhat embarassing and they appear avoidable. We are not troubled by our susceptibility to the vertical-horizontal illusion or by our inability to remember a list of m.ore than eight digits. In contrast, errors of reasoning are often disconcertingfieither becau.se the solution that we failed to find appears quite obvioustin retrospect; or because the error that we made remains attractive although we know it to be an error. Many current studies of judgment are concerned bith problems that have one or the other of these characteristics. $ *This work was supported by the Qffice of Naval Research under Contract NOOci14-79CaO77 ,to ford University. Reprint requests should be sent to Daniel Kahneman, Department of ?sychology, wersity of British Columbia, 2075 Wesbrook hiall, Vancouver B.C., Canada V6T IW5. ;i A 124 iI_ Kahnemaz and A. rrslersky The presence of an error of judgment is iemonstrated by comparing people’s responses either to an established fact (e.g., that the two lines are equal in length) or to an accepted rule of arithmetic, logic or statistics. HOWever, not every response that appears to contradict an established fact or an accepted rule is a judgmental error. The contradiction could also arise from the subject’s misunderstanding of the question, or from the investigator’s misinterpretation of the answer. The der,cription of a particular response as an error of judgment therefore involves assumptions about the communication between the experimenter and the subject. (We shall return to this issue later in the paper.) The student of judgment should avoid overly strict interpretations, which treat reasonable answers as errors, as well as overly charitable interpretations, which attempt to rationalize every response. Although errors of judgment are but a method by which some cognitive processes are studied, the method has become a significant part of the message. The accumulation of demonstrations in which intelligent people violate elementary rules of logic or statistics has raised doubts about the descriptive adequacy of rational models of judgment and decision making. In the two decades following World War II, several descriptive treatments of actual behavior were based on normative models: subjective expected utility theory in analyses of risky choice, the Bayesian calculus in investigations of changes of belief, and signal-detection theory in studies of psychophysical tasks. The theoretical analyses of these situations, and to a much lesser degree the experimental results, suggested an image of people as efficient, nearly optimal decision-makers. On this background, observations of elementary violations of logical or statistical reasoning appeared surprising, and the reaction may have encouraged a view of the human intellect that some authors have criticized as unfairly negative (Cohen, 1979, 1981; Edwards, 1975; Einhorn and Hogarth, 1981). There are three related reasons for the focus on systematic errors and inferential biases in the study of reasoning. First, they expose some of our intellectual limitations and suggest ways of improving the quality aof our thinking,, Second, errors and biases often reveal the psychological processes and the heuristic procedures that govern judgment and inference, Third, mistakes and fallacies help the mapping of human intuitions by indicating which principles of statistics or logic are non-intuitive or counter-intuitive. The terms ‘intuition’ and ‘inttiitive’ are used in three different senses. First, a judgment is called intuitive if it is reached by an informal and unstructured mode of reasoning, without the use of analytic methods or deliberate calculation. For example, most psychologists follow an intuitive procedure in deciding the size of their samples but adopt analytic procedures to test the statistical significance of their results. Second, a formal rule or a fact of nature On the study of statistical Muitions 125 is called intuitive if it is compatible with our lay model of the world. Thus, it is intuitively obvious that the probability of winning a lottery prize decreases with the number of tickets. but it is counter-intuitive that there is a better than even chance that a group of 23 people will include a pair of individuals with the same birthday. T ird, a rule or a procedure is said to be part of our repertoire of intuitions when we apply the rule or follow the procedure in our normal conduct. The rules of &Tammar;, f&r; example, are part of the intuitions of a native spea&er;, and some (though not all) of the rules of plane geometry are incorporated into our spatial reasoning. . The present paper addresses several methodological and conceptual problems that arise in attempts to map people’s intuitions about chance and uncertainty. We begin by discussing different tests of statistical intuitions, we then turn to a critique of the question-answering paradigm in judgment research, and we conclude with a discussion of the non-intuitive character of some statistical laws. Tests of statistical intuitions Errors and biases in judgment under uncertainty are the major source of data for the mapping of the boundaries of people’s statistical intuitions. In this context it is instructive to distinguish between errors of application and errors of comprehension. A failure in a particular problem is called an e&rror; of application if there is evidence that people know and accept a rule that they did not apply. A failure is called an error of comprehension if people d.o not recognize the validity of the rule that they violated. An error of application is most convincingly demonstrated when a person, spontaneously or with minimal prolmpting, clutches his head and exclaims: ‘HOW could I have missed that?’ Although many readers will recognize this experience, such displays of emotion cannot be counted sn, and other procedures must be developed to demonstrate that people understand a rule that they have violated. The understanding of a rule can be tested by ( 1) eliciting from subjects or asking them to endorse, a statement of (2) a general rule or an argument for or against a particular conclusion. The combination of these features yields four procedures, which we now illustrate and discuss. We begin with an informal example in which understanding of a rule is confirmed by the acceptance or end’orsement of an argument. One of us has presented the following question to many squash players: ‘As you know, a game of squash can bplayed either to 9 or to 15 points. Holding all other rules of the game constant, it A is a better player than B, whtch sccring system will give A a better chance of winning?’ 126 D. Kahneman and A. Tversky Although all our informants had some knowledge of statistics, most of them said that zhe sc0rin.g system should not make any difference. They were then asked to consider the argument that the better player should prefer the longer game, because an atypical outcome is less likely to occur in a large sample than in a small one. With very few exceptions, the respondents immediately accepted the argument, and admitted that their initial response had been a mistake. Evidently, our informants had some appreciation of the effect of sample size on sampling errors, but they failed to code the length of a squash game as an instance of sample size. The fact that the correct conclusion becomes compelling as soon as this connection is made indicates that the initial response was an error of application, not of comprehension. A more systematic attempt to diagnose the nature of an error was made in a study of a phenomenon labelled the conjunction effect (Tversky and Kahneman, in press). Perhaps the most elementary principle of probability theory is the conjunction rule, which states that the probability of a conjunction A&B; cannot exceed either the probability of A or the probability of B. As the following example shows, however, it is possible to construct tests in which most judges-even high!jl sophisticated ones-state that a conjunction of events is more probable than one of its components. To induce the conjunction effect, we presented subjects with personality sketches of the type illustrated below: ‘Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations.’ In one version of the problem, respondents were asked which of two statements about Linda was more probable: A. Linda is a bank teller; B. Linda is a bank teller who is active in the femin.i?t movement. It? 3 large sample of statistically na’ive undergraduates, 86% judged the second statement to be more probable. In a sample of psychology graduate students, only 50% committed this error. However, the difference between statistically naive and sophisticated respondents vanished when the two critical items were embedded in a list of eight comparable statements about Linda. Over 80% of both groups exhibited the conjunction effect. Similar results were obtained in a between-subject design, in which the critical categories were compared indirectly (Tversky and Kahneman, in press). Tests of rule-endorsement and argument-endorsement were used in an effort to determine whether people understand and accept the conjunction rule. First, we presented a abiPIty of X On the study of statistical intuitions I27 and Y’ was endorsed by 81% of the respondents. In comparison, only 6% endorsed ‘If A is more probable than B, then they cannot both occur’. Thesca results indicate some understanding of the conjunction rule, although the en-dorsement is not unanimous, perhaps because of the abstract and unfamiliar formulation. An argument-endorsement procedure was also employed in which respondents were given the description of Linda, followed by statements A and B above and were asked to check which of the following arguments they con-